“Truthful” acting emerges through forward model development

Open peer commentary on the article ““Black Box” Theatre: Second-Order Cybernetics and Naturalism in Rehearsal and Performance” by Tom Scholte. Upshot: My aim is to show that “truthful” acting that emerges through improvisation is equivalent to the development of mutual forward models in the actors. If these models match those of the audience members, this is perceived as “truthful.

Modelling the Effect of Dorsal Raphe Serotonin Neurons on Patience for Future Rewards

Serotonin is a neurotransmitter that is implicated in many basic human functions and behaviours and is closely associated with happiness, depression and reward processing. In particular it appears to be involved in suppressing responses to distracting stimuli while waiting for a delayed reward. Here we present a system level model of the limbic system which is able to generate a serotonin (5-hydroxytryptamine [5HT]) signal so that a simulated animal waits for a delayed reward. We propose that the 5HT signal is computed by a network involving the medial Orbital Frontal Cortex (mOFC), medial Pre Frontal Cortex (mPFC), Dorsal Raphe Nucleus (DRN)and the Nucleus Accumbens Core (NAcc). The serotonin signal encodes pre-reward liking, motivation throughout the trial and delayed reward waiting. We have successfully replicated the behaviour and dynamics of laboratory studies. With the help of this model we can predict that low levels of serotonin indirectly cause less encountered rewards because the animal gives up too early

Sequence-learning in a self-referential closed-loop behavioural system

This thesis focuses on the problem of "autonomous agents". It is assumed that such agents want to be in a desired state which can be assessed by the agent itself when it observes the consequences of its own actions. Therefore the feedback from the motor output via the environment to the sensor input is an essential component of such a system. As a consequence an agent is defined in this thesis as a self-referential system which operates within a closed sensor- mot or-sensor feedback loop. The generic situation is that the agent is always prone to unpredictable disturbances which arrive from the outside, i.e. from its environment. These disturbances cause a deviation from the desired state (for example the organism is attacked unexpectedly or the temperature in the environment changes, ...). The simplest mechanism for managing such disturbances in an organism is to employ a reflex loop which essentially establishes reactive behaviour. Reflex loops are directly related to closed loop feedback controllers. Thus, they are robust and they do not need a built-in model of the control situation. However, reflexes have one main disadvantage, namely that they always occur "too late"; i.e., only after a (for example, unpleasant) reflex eliciting sensor event has occurred. This defines an objective problem for the organism. This thesis provides a solution to this problem which is called Isotropic Sequence Order (ISO-) learning. The problem is solved by correlating the primary reflex and a predictive sensor input: the result is that the system learns the temporal relation between the primary reflex and the earlier sensor input and creates a new predictive reflex. This (new) predictive reflex does not have the disadvantage of the primary reflex, namely of always being too late. As a consequence the agent is able to maintain its desired input-state all the time. In terms of engineering this means that ISO learning solves the inverse controller problem for the reflex, which is mathematically proven in this thesis. Summarising, this means that the organism starts as a reactive system and learning turns the system into a pro-active system. It will be demonstrated by a real robot experiment that ISO learning can successfully learn to solve the classical obstacle avoidance task without external intervention (like rewards). In this experiment the robot has to correlate a reflex (retraction after collision) with signals of range finders (turn before the collision). After successful learning the robot generates a turning reaction before it bumps into an obstacle. Additionally it will be shown that the learning goal of "reflex avoidance" can also, paradoxically, be used to solve an attraction task

Model checking learning agent systems using Promela with embedded C code and abstraction

As autonomous systems become more prevalent, methods for their verification will become more widely used. Model checking is a formal verification technique that can help ensure the safety of autonomous systems, but in most cases it cannot be applied by novices, or in its straight \off-the-shelf" form. In order to be more widely applicable it is crucial that more sophisticated techniques are used, and are presented in a way that is reproducible by engineers and verifiers alike. In this paper we demonstrate in detail two techniques that are used to increase the power of model checking using the model checker SPIN. The first of these is the use of embedded C code within Promela specifications, in order to accurately re ect robot movement. The second is to use abstraction together with a simulation relation to allow us to verify multiple environments simultaneously. We apply these techniques to a fairly simple system in which a robot moves about a fixed circular environment and learns to avoid obstacles. The learning algorithm is inspired by the way that insects learn to avoid obstacles in response to pain signals received from their antennae. Crucially, we prove that our abstraction is sound for our example system { a step that is often omitted but is vital if formal verification is to be widely accepted as a useful and meaningful approach

Sign and Relevance learning

Standard models of biologically realistic, or inspired, reinforcement learning employ a global error signal which implies shallow networks. However, on the other hand, local learning rules allow networks with multiple layers. Here, we present a network combining local learning with global modulation where neuromodulation controls the amount of plasticity change in the whole network, while the sign of the error is passed via a bottom-up pathway through the network. Neuromodulation can be understood as a rectified error, or relevance, signal while the bottom-up sign of the error signal decides between long-term potentiation and long-term depression. We demonstrate the performance of this paradigm with a real robotic task as a proof of concept.Comment: 26 pages, 12 figure

High precision ECG Database with annotated R peaks, recorded and filmed under realistic conditions

This database contains ECGs from 25 subjects. Each subject was recorded performing 5 different tasks for two minutes: •sitting •a maths test on a tablet •walking on a treadmill •running on a treadmill •using a hand bike The following channels were recorded with two Attys running synchronously: •Einthoven II and III with standard cables and the amplifier worn around the waist •Exercise cheststrap ECG which resembles approximtely V2-V1 with the ECG amplifier directly mounted on the strap •Acceleration in X/Y/Z whith the sensor mounted directly on the chest strap The cheststrap ECG allowed R peak detection even while jogging at a very high precision (+/- one sample). The sampling rate was 250Hz at a resolution of 24 bits. The database contains the unfiltered, DC-coupled signals as originally recorded. In order to be able to link the ECG artefacts to the behaviour of the subject all but one subject gave permission to be filmed and the videos are also part of the database

Closed-loop deep learning: generating forward models with back-propagation

A reflex is a simple closed loop control approach which tries to minimise an error but fails to do so because it will always react too late. An adaptive algorithm can use this error to learn a forward model with the help of predictive cues. For example a driver learns to improve their steering by looking ahead to avoid steering in the last minute. In order to process complex cues such as the road ahead deep learning is a natural choice. However, this is usually only achieved indirectly by employing deep reinforcement learning having a discrete state space. Here, we show how this can be directly achieved by embedding deep learning into a closed loop system and preserving its continuous processing. We show specifically how error back-propagation can be achieved in z-space and in general how gradient based approaches can be analysed in such closed loop scenarios. The performance of this learning paradigm is demonstrated using a line-follower both in simulation and on a real robot that show very fast and continuous learning.Comment: 13 pages, 6 figure

A functional electrical stimulation system for human walking inspired by reflexive control principles

This study presents an innovative multichannel functional electrical stimulation gait-assist system which employs a well-established purely reflexive control algorithm, previously tested in a series of bipedal walking robots. In these robots, ground contact information was used to activate motors in the legs, generating a gait cycle similar to that of humans. Rather than developing a sophisticated closed-loop functional electrical stimulation control strategy for stepping, we have instead utilised our simple reflexive model where muscle activation is induced through transfer functions which translate sensory signals, predominantly ground contact information, into motor actions. The functionality of the functional electrical stimulation system was tested by analysis of the gait function of seven healthy volunteers during functional electrical stimulation–assisted treadmill walking compared to unassisted walking. The results demonstrated that the system was successful in synchronising muscle activation throughout the gait cycle and was able to promote functional hip and ankle movements. Overall, the study demonstrates the potential of human-inspired robotic systems in the design of assistive devices for bipedal walking

Measuring vowel percepts in human listeners with behavioral response-triggered averaging

A vowel can be largely defined by the frequencies of its first two formants, but the absolute frequencies for a given vowel vary from talker to talker and utterance to utterance. Given this variability, it is unclear what criteria listeners use to identify vowels. To estimate the vowel features for which people listen, we adapted a noise-based reverse-correlation method from auditory neurophysiological studies and vision research (Gold et al., 1999). Listeners presented with the stimulus, which had a random spectrum with levels in 60 frequency bins changing every 0.5 s, were asked to press a key whenever they heard the vowels [a] or [i:]. Reverse-correlation was used to average the spectrum of the noise prior to each key press, thus estimating the features of the vowels for which the participants were listening. The formant frequencies of these reverse-correlated vowels were similar to those of their respective whispered vowels. The success of this response-triggered technique suggests that it may prove useful for estimating other internal representations, including perceptual phenomena like tinnitus. References: Gold, J., Bennett, P. J., and Sekuler, A. B. (1999). “Identification of band-pass filtered faces and letters by human and ideal observers,” Vis. Res. 39(21), 3537–3560

Technical developments of functional electrical stimulation to restore gait functions : sensing, control strategies, and current commercial systems

The work presents a review on the technological advancements of functional electrical stimulation (FES) neuroprostheses to restore gait walking over the last decades. The aim of an FES intervention is to functionally restore and rehabilitate individuals with motor disorders, such as stroke, spinal cord injury, multiple sclerosis, and others. The technique has been applied for widespread practical use for several years due to the rapid development of micro- and nano-technology. This technical review covers neuroprostheses developed within academia and currently available on the market. These systems are thoroughly analyzed and discussed with particular emphasis on the sensing techniques and control strategies. In the last part, a combination of FES technology and exoskeletons is presented as an emerging solution to overcome the drawbacks of current FES-based neuroprostheses, and recommendations on future research direction are suggeste